Skip to content

Comments

Introduce Telemetry Processor#987

Merged
solnic merged 2 commits intomasterfrom
solnic/telemetry-processor-logs
Feb 18, 2026
Merged

Introduce Telemetry Processor#987
solnic merged 2 commits intomasterfrom
solnic/telemetry-processor-logs

Conversation

@solnic
Copy link
Collaborator

@solnic solnic commented Feb 9, 2026

⚠️ Proposed plan for 12.0.0 release

  1. Structured logging included using Telemetry Processor by default
  2. Remaining categories (:errors, :check_ins, :transactions) can be enabled for Telemetry Processor using a new option called telemetry_processor_categories - I'll add those in separate PRs
  3. At some point original sender gets deprecated
  4. In 13.0.0 we replace sender with telemetry processor

Summary

Introduce the TelemetryProcessor — an OTP Supervisor that buffers, batches, and ships log events to Sentry asynchronously, implementing the Backend Telemetry Processor spec.

This is step 1 and currently only the :log category is routed through it. Errors, transactions, and check-ins continue using existing transport paths.

Architecture

  • Buffer — Fixed-capacity FIFO queue (1000 items), drops oldest on overflow
  • Scheduler — Signal-based drain coordination with weighted round-robin priority and integrated bounded transport queue (1000 items), sequential HTTP sends via spawn_monitor
  • TelemetryProcessor — Supervisor owning Buffer + Scheduler, exposes add/2 and flush/0

Data flow

flowchart TD
    L["Logger"] --> LB["LogsBackend"]
    LB --> TP["TelemetryProcessor.add/2"]
    TP -->|cast| B["Buffer (1000 items)"]
    TP -->|cast| S["Scheduler"]
    S -->|poll| B
    B -->|batch| S
    S -->|spawn_monitor| HTTP["HTTP POST to Sentry"]
Loading

@solnic solnic force-pushed the solnic/telemetry-processor-logs branch 11 times, most recently from 897eb10 to 1fb8bc7 Compare February 12, 2026 10:20
@solnic solnic changed the title WIP: introduce TelemetryProcessor for Logs Introduce Telemetry Processor Feb 12, 2026
@solnic solnic marked this pull request as ready for review February 12, 2026 11:35
@dingsdax dingsdax requested a review from giortzisg February 12, 2026 12:50

maybe_log_event_buffer =
maybe_telemetry_processor =
if Config.enable_logs?() do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not forwards all data?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dingsdax this is done in the subsequent PRs that are waiting in line :)

Copy link
Contributor

@dingsdax dingsdax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👍

@solnic solnic force-pushed the solnic/telemetry-processor-logs branch from 1fb8bc7 to 6577f75 Compare February 18, 2026 12:43
@solnic solnic force-pushed the solnic/telemetry-processor-logs branch from 6577f75 to 95975a9 Compare February 18, 2026 13:04
@solnic solnic force-pushed the solnic/telemetry-processor-logs branch from 95975a9 to c081ca3 Compare February 18, 2026 13:05
Comment on lines +46 to +56
defstruct [
:buffers,
:priority_cycle,
:cycle_position,
:on_envelope,
:capacity,
:active_ref,
:active_item_count,
queue: :queue.new(),
size: 0
]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The Scheduler struct field active_item_count is not initialized, defaulting to nil. This can cause a crash when used in an arithmetic operation.
Severity: MEDIUM

Suggested Fix

Initialize active_item_count to 0 in the defstruct definition. This ensures the field always holds a valid integer, preventing potential ArithmeticError crashes when it's used in calculations.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: lib/sentry/telemetry/scheduler.ex#L46-L56

Potential issue: The `Scheduler` struct defines the `:active_item_count` field without a
default value, causing it to be initialized to `nil`. This field is later used in a
subtraction operation within the `handle_info/2` function when processing `:DOWN`
messages. If a `:DOWN` message is received before an envelope is processed and
`active_item_count` is set to an integer, the `Scheduler` process will crash with an
`ArithmeticError`. While the normal execution path may not trigger this, it's an unsafe
state that violates the field's typespec and can be triggered in edge cases or by future
code changes.

Copy link
Member

@sl0thentr0py sl0thentr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work!

@solnic solnic merged commit d63a163 into master Feb 18, 2026
11 checks passed
@solnic solnic deleted the solnic/telemetry-processor-logs branch February 18, 2026 13:19
Copy link
Member

@philipphofmann philipphofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late post merge review, and disclaimer: I have almost zero experience with Elixir, so take this with a grain of salt.

This is optional: if I'm not mistaken, the rate limits are only checked in the transport. You could already drop rate-limited items before adding them to the buffer. And you could drop rate-limited items in the scheduler before you pass them down to the transport. These points are optional, and I leave it up to you if it's worth the effort to add these.

The rest looks good to me, and thanks for adding this.

Comment on lines +177 to +181
defp offer(%Buffer{size: size, capacity: capacity} = state, item)
when size >= capacity do
{{:value, _dropped}, items} = :queue.out(state.items)
%{state | items: :queue.in(item, items)}
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow up comment: if I'm not mistaken, this looks like we're dropping data here, and if we do, we must record client reports: https://develop.sentry.dev/sdk/foundations/processing/telemetry-processor/#telemetry-buffer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants